CooperateISO by adrianhutter · Pull Request #1498 · Axelrod-Python/Axelrod

adrianhutter · 2026-08-05T11:33:12Z

Adding 3 strategies (LongtermTfT, ISO, CooperateISO) from https://arxiv.org/abs/2303.03519.

Made a few changes relative to the versions used in the paper. Most importantly, ISO now uses scipy.optimize instead of torch's Adam. This works equally well, is faster, and avoids having to import torch.

Ran the CooperateISO implementation from this PR against all strategies in Axelrod-4.14.0, and verified that it outperforms EvolvedLookerUp2_2_2 and DBS at both noise levels 0% and 10%.

Fix Read the Docs build failures by updating documentation requirements and conf.py:

Pinned sphinx to <9.0.0 and added sphinx-rtd-theme in docs/requirements.txt to ensure compatibility and prevent theme lookup errors.
Removed scipy from the mock modules list in docs/conf.py so that submodules like scipy.optimize can be correctly imported from the installed package during documentation generation.

drvinceknight

This looks really great, I enjoyed the preprint (congrats on the work)!

My requests are mainly stylistic as we tend to try and avoid inline comments (there are a number in the tests that I didn't comment on that could perhaps be moved to docstrings).

drvinceknight · 2026-08-07T08:34:24Z

@@ -1,4 +1,7 @@
+sphinx>=7.0.0,<9.0.0


Can you help me understand why this change is needed? (This might be because of an upstream change.)

drvinceknight · 2026-08-07T08:35:02Z


 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
-on_rtd = os.environ.get("READTHEDOCS", None) == "True"


Could you help me understand why this change? (It might be something got stale that I'm not remembering.)

drvinceknight · 2026-08-07T08:35:46Z

+            return opponent.history[-1]
+
+
+# We describe memory-1 strategies as length-4 arrays, quantifying the probability of cooperation in the states [CC, CD, DC, DD].


Suggested change

# We describe memory-1 strategies as length-4 arrays, quantifying the probability of cooperation in the states [CC, CD, DC, DD].

drvinceknight · 2026-08-07T08:36:17Z

+    # Apply p_noise only to own strategy, not to opponent
+    # (the opponent strategy already includes noise effects).
+    own = my_strategy + p_noise * (1.0 - 2.0 * my_strategy)
+
+    # Flip CD/DC for opponent.
+    opp = opp_strategy[[0, 2, 1, 3]]
+
+    # Build the transition matrix.
+    trans_mat = np.array(
+        [
+            own * opp,
+            own * (1.0 - opp),
+            (1.0 - own) * opp,
+            (1.0 - own) * (1.0 - opp),
+        ]
+    ).T
+
+    R, P, S, T = RPST
+    rewards = np.array([R, S, T, P], dtype=float)
+
+    # Don't include init state in summed rewards.
+    inv = np.linalg.inv(np.eye(4) - (1.0 - p_end) * trans_mat)
+
+    # Calculate expected reward,
+    reward = init_state @ (inv @ rewards - rewards)
+
+    # Avg. reward per step
+    return p_end * float(reward) / (1.0 - p_end)


Could you remove this inline comments and put the corresponding explanation in the docstring please.

drvinceknight · 2026-08-07T08:36:40Z

+    # Clamp to possible values, given noise
+    opp = np.clip(opponent, p_noise, 1.0 - p_noise)
+
+    # Setup initial state
+    init_state = np.zeros(4, dtype=np.float32)
+    init_state[init_state_idx] = 1.0
+
+    # Define the objective function to minimize (negative reward)
+    def objective(params: np.ndarray) -> float:
+        return -get_reward(params, opp, init_state, p_end, p_noise, RPST)
+
+    x0 = np.array([0.5, 0.5, 0.5, 0.5])
+    bounds = [(0.0, 1.0), (0.0, 1.0), (0.0, 1.0), (0.0, 1.0)]
+    result = minimize(
+        objective, x0, method="L-BFGS-B", bounds=bounds, options={"maxiter": 50}
+    )
+
+    # result.fun is the minimum loss (-reward), result.x are the optimal parameters
+    return -result.fun, result.x


I think we can remove the inline comments here

drvinceknight · 2026-08-07T08:37:10Z

+        # Track the opponent's rate of cooperation (numerator, denominator) for each state.
+        # Assume we have seen the opponent play following TfT once in each state,
+        # to make the opponent-model well-defined from the start.
+        self.ewma_CC = [1.0, 1.0]
+        self.ewma_CD = [1.0, 1.0]
+        self.ewma_DC = [0.0, 1.0]
+        self.ewma_DD = [0.0, 1.0]
+
+        # Initial cooperation probabilities (num / den)
+        self.opp_model = [1.0, 0.0, 1.0, 0.0]
+        self.my_policy = [1.0, 0.0, 1.0, 0.0]


Can you move these inline comments to the docstring of the __init__

Co-authored-by: Vince Knight <vince@vknight.org>

adrianhutter · 2026-09-03T21:10:49Z

@drvinceknight Thanks a lot for review. I addressed all comments, please take another look.

marcharper · 2026-10-12T22:06:49Z

+            self.opp_model[0] = pr_c
+        elif prev_state == (C, D):
+            pr_c = self._update_single_ewma(self.ewma_CD, opp_act)
+            self.opp_model[2] = pr_c


Just checking that the index of 2 is correct here, since the ordering is different than the ewma fields and in the _get_state_idx function (also why is _get_state_idx not used here?).

It might be clearer to use dictionaries with the prev_state tuples as keys for both opp_model and ewma_**.

marcharper · 2026-10-12T22:07:04Z

+            self.opp_model[2] = pr_c
+        elif prev_state == (D, C):
+            pr_c = self._update_single_ewma(self.ewma_DC, opp_act)
+            self.opp_model[1] = pr_c


marcharper · 2026-10-12T22:10:46Z

+
+        Returns our expected reward per step.
+        """
+        self._update_opponent_model(opponent)


If opponent.history is all that's used by these functions consider passing just that instead of the entire object

marcharper · 2026-10-12T22:10:54Z

+        Returns our expected reward per step.
+        """
+        self._update_opponent_model(opponent)
+        state_idx = self._get_state_idx(opponent)


marcharper · 2026-10-12T22:26:22Z

+        self.noise = self.match_attributes["noise"]
+        self.iso_instance.noise = self.noise
+
+    def _update_reward_history(self, opponent):


Could simplify by computing a dict of (action, action) --> reward in __init__, then this is just self.reward_history.append(reward_dict[state]). This eliminates the repeated unpackings and if/elif chain.

marcharper · 2026-10-12T22:34:27Z

        players = [axl.Cooperator(), axl.Defector()]
-        tournament = axl.Tournament(players, turns=5, repetitions=3, noise=0.75)
+        tournament = axl.Tournament(
+            players, turns=5, repetitions=3, noise=0.75, seed=42


Was this failing sometimes? Otherwise why set the seed?

marcharper · 2026-10-12T22:35:25Z

+            (C, D),
+            (
+                C,
+                D,


Please remove the extra comma (and just below) so that the formatting is consistent

marcharper · 2026-10-12T22:41:11Z

+            return C
+        if len(self.history) == 1:
+            return opponent.history[-1]
+        if self.history[-2] == C:


What's the reason for only updating z and n_tft_would_c on cooperation?

marcharper · 2026-10-12T22:44:07Z

+    rewards = np.array([R, S, T, P], dtype=float)
+    inv = np.linalg.inv(np.eye(4) - (1.0 - p_end) * trans_mat)
+    reward = init_state @ (inv @ rewards - rewards)
+    return p_end * float(reward) / (1.0 - p_end)


Does this function break if p_end == 1 or p_noise == 1?

marcharper · 2026-10-12T22:46:26Z

+    Discovers the optimal response strategy (policy) against a fixed opponent
+    model by maximizing the expected reward from a given starting state.
+    """
+    assert p_noise < 0.5


Let's avoid the assert here so we don't break runs with p_noise >= 0.5

adrianhutter added 19 commits July 10, 2026 15:27

Add bibliography reference

892cdf4

LongtermTfT

96deb3d

ISO

3b9f0a1

Cosmetics

6a83d92

Add tests for LongtermTFT and ISO

20b4d52

Fixes

b36a110

Implement CooperateISO

cd51442

Test for CooperateISO

ab25e10

Switch optimizer from torch to scipty

813b441

Add ISO tests against specific strategies

e7dabb9

Refine tests

68a3781

Refine tests

ddee85c

Update comments

79d068d

Remove debug prints

2e0c252

Remove redundant test

63269bb

Add CooperateISO to _strategies

06b7f2c

Comments

3567789

Remove obsolete test

63aadf3

Format code with Black

6889fdf

adrianhutter marked this pull request as ready for review August 5, 2026 12:15

adrianhutter added 10 commits August 5, 2026 12:19

Fix import ordering with isort

56a926d

Trigger CI

06628f8

Add strategies to all_strategies

1636a7c

Add to strategy_index.rst

752a2f6

Add ISO and CooperateISO to long_run_strategies

c4bbfe2

Update strategy counts

2801437

Achieve 100% test coverage

3e779d6

Formatting

1bccc4d

Add scipy to requirements

423ccbc

Fix sphinx issues

5948f2f

adrianhutter added 2 commits August 5, 2026 15:28

Remove scipy from MOCK_MODULES

f291701

Update number of long-running strategies

2ac1329

drvinceknight requested changes Aug 7, 2026

View reviewed changes

adrianhutter and others added 8 commits August 8, 2026 11:05

Apply suggestion from @drvinceknight

7052ed4

Co-authored-by: Vince Knight <vince@vknight.org>

Remove inline comments

062faa4

Remove inline comments

3ac7bd1

Remove inline comments and sphinx pin

06dd733

black formatting

3b775fd

Make test of noisy IPD deterministic

d2f07c7

Fix seeding

ee0234f

black formatting

ae85352

adrianhutter requested a review from drvinceknight August 17, 2026 20:36

drvinceknight requested a review from marcharper October 12, 2026 14:27

marcharper requested changes Oct 12, 2026

View reviewed changes

		return opponent.history[-1]


		# We describe memory-1 strategies as length-4 arrays, quantifying the probability of cooperation in the states [CC, CD, DC, DD].

Conversation

adrianhutter commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

drvinceknight left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

adrianhutter commented Sep 3, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

adrianhutter commented Aug 5, 2026 •

edited

Loading